1. Field
The present technology relates to systems and methods of encoding and decoding audio, video and other multimedia data. More particularly, the technology relates to computer architecture and operating methods that can allow for reconfiguration of a multimedia decoder without interrupting multimedia playback.
2. Description of the Related Art
Digital multimedia capabilities can be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless communication devices such as radio telephone handsets, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, digital cameras, digital recording devices, video gaming devices, video game consoles, and the like. Digital multimedia devices implement video encoding techniques, such as MPEG-2, MPEG-4, or H.264/MPEG-4, Part 10, Advanced Video Coding (AVC), in order to store, transmit, and receive digital video efficiently. Digital devices implement audio encoding techniques or formats such as AAC, MP3, and WAV to store, transmit, and receive digital audio efficiently. Digital devices further implement additional data and graphics encoding techniques or formats such as IGES, 3DT, PS, MNG ODF and SVG
Encoding is predominantly proprietary or standards-based. Accordingly, multimedia is generally encoded in a fixed format. While current methods may provide some adaptation to changes in characteristics such as content, this adaptation is limited, e.g., in the case of video to the selection between a handful of modes in H.264/AVC. In existing protocols, there is no true content adaptation; once a standard is fixed, modification to the coding algorithms or data tables is not permitted. Further, devices that incorporate digital multimedia capabilities generally must conform to these standards. As a result, these devices have fixed decoders that are configured to decode only multimedia encoded in the fixed format(s). Selection between pre-determined formats, e.g. MPEG-2, MPEG-4, and H.263, has been suggested as in U.S. Pat. No. 7,085,320 and US Patent Application 2003/0182544. Similarly, utilization and selection between scalable coding layers within standards at the encoder was suggested in U.S. Pat. No. 6,553,072 and a corresponding decoder proposed. In these cases the utilized formats and their components are known at the encoder and the decoder. Hence there is still no true content adaptation, since the characteristics of the coding algorithms are fixed by the pre-determined coding formats.
In certain instances, it would be beneficial to utilize encoding or decoding algorithms generated based on the content to be encoded or parameters of the system, as doing so may more efficiently encode or decode multimedia data. However, decoders in devices for receiving such encoded multimedia data may not be configured to decode content encoded using such generated encoding methods. This lack of configuration may be due to reasons such as a limit on the number of decoders able to be implemented in a device, no support for legacy decoders, and no support for new encoders. Solutions to this problem have been explored by permitting either selection between sub-algorithms or codec components or reconfiguration of portions of the decoder. FIG. 1(a) illustrates selection between known decoder sub-algorithms or elements. Various selection strategies and targets have been suggested, such as variable-length decoder tables as in US Patent Application 2004/0028141 or code rates, interleaving techniques, and error control techniques as in U.S. Pat. No. 5,666,170. Alternatively, FIG. 1(b) illustrates reconfiguration of decoder elements in hardware or software in response to control signals. Reconfiguration can be performed via methods such as dynamically interconnecting elements such as in US Patent Application 2003/0182544, US Patent Application 2011/0032985, and US Patent Application 2008/0130742. Further reconfiguration methods are to dynamically reconfigure portions of decoder processes or elements, such as adder arrays as in US Patent Application 2008/0130742 and caches as in US Patent Application 2007/0064006. In U.S. Pat. No. 7,385,532 a purely hardware decoder is reconfigured in response to bitstream commands.
Alternatively, software configuration or reconfiguration of portions of the decoder in response to configuration commands has been proposed. These have been predominantly toolbox-based. FIG. 2 shows a system from U.S. Pat. No. 5,987,181 which illustrates that a set of tools is known to both the encoder and decoder each of which can draw upon this set. The choice of tools used at the encoder is then transmitted to the decoder and the decoder reconfigures based upon the tools in the local toolbox. A similar solution was proposed in O. Avaro, P. A. Chou, A. Eleftheriadis, C. Herpel, C. Reader and J. Signès, “The MPEG-4 Systems and Description Language: A Way Ahead in Audio Visual Information Representation” and then more recently implemented in the MPEG Reconfigurable Video Coding (RVC) initiative as discussed in S. Lee, E. S. Jang, M. Mattavelli, C.-J. Tsai, “Working Draft 5 of ISO/IEC 23001-4: Codec Configuration Representation”, ISO/IEC JTCI/SC20/WG11 document N9223 and E. S. Jang, J. Ohm, and M. Mattavelli, “Whitepaper on Reconfigurable Video Coding (RVC)”, ISO/IEC JTCI/SC20/WG11 document N9586. The RVC decoder solution is illustrated in FIG. 3. Although only the decoder will be specified when then standard is developed, a set of toolboxes is available to both the encoder and decoder. The decoder constructs a model from the indicated tools, called functional units, by accessing the toolbox, and then instantiates a decoder by linking the selected functional units to the current decoder. This use of set toolboxes or tool libraries is further discussed in US Patent Application 2011/0032985 and well as in U.S. Pat. No. 7,385,832.
Reconfiguring a single sub-algorithm without the use of a toolbox at the decoder was proposed in M. Bystrom, I. Richardson, S. Kannangara, and M. de-Frutos-Lopez, “Dynamic Replacement of Video Coding Elements” and in C. S. Kannangara, J. Philp, I. Richardson, M. Bystrom, and M. de-Frutos-Lopez, “A Syntax for Defining, Communicating, and Implementing Video Decoder Function and Structure”. In these cases, the single element, a backwards transform, was encoded and transmitted to the decoder where it replaced the existing backwards transform either through the well-known binary patching method or code replacement and just-in-time compilation.